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Abstract 



We have searched a sample of 9.6 million BB events for the lepton- flavor- 
violating decays B he^^"^, B^ —>■ h~e^e^, B^ h~e'^ii'^, and B~^ 
h~ iJ,~^H~^, where h is vr, K, p, and isr*(892), a total of sixteen modes. We find 
no evidence for these decays, and place 90% confidence level upper limits on 
their branching fractions that range from 1.0 to 8.3 x 10^^. 
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The Standard Model predicts that the branching fractions for the decays b — > se~^e~ and 
b sfj,^iJ,~ will be small but non-zero, of order 10^^. We have previously conducted searches for 
those inclusive decays |l] and also for the exclusive decays B — > Ki^i^ and B —>■ K*{892)£^i^ 
that would result from the quark-level processes. Others |0-|6| have also searched for the exclusive 
decays. Upper limits are now close to the Standard Model predictions, and there is evidence for 
B Ki+l- |]. 

In contrast, the Standard Model predicts that the topologically similar, but lepton-flavor- 
violating decays h —>■ se^fj.^ and b de^fj.^ vanish identically, as do the decays B~^ X~i'^£^ 
and B^ — s- X^i'^i'^. These decays are predicted to occur in many theories "beyond the Standard 
Model", for example multi-Higgs extensions 0, theories with leptoquarks and theories with 
Majorana neutrinos |^. The recent evidence |jl^ that neutrinos mix, and therefore have mass, while 
not leading to predictions of observable rates for lepton-flavor-violating decays involving charged 



leptons, nonetheless heightens interest in them, as does the recent claim [11| of neutrinoless double 
beta decay. 

While the underlying physics of lepton-flavor-violating decays is very different from that of those 
decays mentioned in the first paragraph, the experimental approach in searching for them is quite 
similar. We have therefore used the techniques described in Ref. ^ to search forQ B — s- Ke^iJ^, 
B K*e^fi^, B —5- vre^/x^, and B —>■ pe^jj^, and also for B^ —>■ /i~e+e^, h~e~^p'^, and fx^jj,^ , 
where h~ is K^, K*~ , vr", and . We have previously |l| searched for the inclusive decay b — 
se^p^, obtaining a 90% confidence level upper limit B{b se^ p~) + B{b — > se~ p,'^) < 2.2 x 10~^ . 
The BaBar collaboration has also searched for, and reported limits on, the related exclusive 
decays, B{B+ K^e^p^) < 0.8 x lO'^ , B{B^ K^e^p^) < 4.1 x 10"^ , B{B+ K*+e^p^) < 
8.0 X 10-^ , and B{B^ K*^e^p^) < 3.3 x 10^^ . 

The data used in this analysis were taken with the CLEO detector at the Cornell Electron 
Storage Ring (CESR), a symmetric e~^e~ collider operating in the T(4S) resonance region. The 
data sample consists of 9.2 fb~^ at the resonance, corresponding to 9.6 million BB events, and 
4.5 fb~^ at a center-of-mass energy 60 MeV below the resonance. The sample below the resonance 
provides information on the background from continuum processes e~^e~ qq, q = u,d,s,c, and 
was used as a check on our Monte Carlo simulation of this background. 

Summing over e^p~ and e~ p~^ , we search for B — > Ke^p^ in both the and modes, 
and for B K*e^p'^ in the K+n' and K^tt^ modes and in the K*^ ^ K^n^ and K^tt^ 

modes, a total of 6 experimentally distinct final states. (Throughout this article, charge conjugate 
modes are implied.) Similarly, we search for B Tre^p^ in both the vr^ and vr*^ modes, and for 
B pe^p^ in both the p^ — > Tr^vr*^ and p^ vr^vr" modes, 4 distinct final states. In the like-sign 
search B~^ h~£~^£~^, we search for five hadronic final states {h~ = K~; vr"; K*~ K~7:^, K^it~; 



^Throughout this article, the symbol K* means K*{892). 
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and p~ it~t:^) for each of e^e"*", e^/x"'", and fi^fi~^, 15 distinct modes. The candidates are 
detected via the — > — > TT~^Tr~ decay chain; vr*^ candidates via vr*^ 77. 

For those decay modes involving a charged kaon, we use specific ionization {dE/dx) and time- 
of-flight information to identify the kaon, cutting loosely (3 standard deviations) if those variables 
deviate from the mean for kaons in the direction away from the mean for pions, and cutting harder 
(1.5 to 2.2 standard deviations, depending on mode) if they deviate on the side towards the pions. 

There are three main sources of backgroundfl B ijj^'^ £+£', and other B ijj^'^X 

decays; BB decays other than B — > ip^'^X, with two apparent leptons (either real leptons or hadrons 
misidentified as leptons); and continuum processes with two apparent leptons. 

In our previous search j^, for B K^*^e'^e~ and B — > K^*^ fj,~^ , the backgrounds from ip and 
ip' were severe. In the searches reported here they are much less of a problem, appearing only when 
particles are misidentified. Examples are B~ — > K~ip, ip e~*"e~, with the K~ misidentified as a 
li~ , and the e~ misidentified as a K~; B" K^tp, ip with the K~ misidentified as , 

and the /x"*" misidentified as tt~^; S~ — s- K'tp, tp — > e~^e~, with one of the identified as /i^. To 
reduce these backgrounds, we required that a lepton candidate that passes identification criteria 
both for and fi^ only be considered as an electron candidate. Also, we discarded a candidate 
reconstruction if any oppositely-charged hadron-lepton pair, if interpreted as a lepton-lepton pair, 
had a pair mass within 30 MeV of ip or ip' mass, or if the e^//^ pair, if interpreted either as e'^e~ or 
H~^IJ,~ , had a pair mass within 50 MeV of ip or 40 MeV of ip'. With these requirements, backgrounds 
from ip and ip' were rendered negligible, less than 0.1 event per decay mode. 

We discriminate between signal events and the remaining two background sources using an 
unbinned maximum likelihood method, including four variables in the likelihood function. (We 
select events for consideration by first applying loose cuts in those variables.) To help distinguish 
between signal and the background from BB semileptonic decays, we use the event missing energy, 
£"111188, since events with leptons from semileptonic B or D decay contain neutrinos, and thus will 
have missing energy. We apply loose cuts, —2.0 < -Emiss < +2.0 GeV. To help distinguish between 
signal and continuum events, we use a Fisher discriminant, a linear combination of R2 (the ratio of 
second and zeroth Fox- Wolfram moments |l^ of the event), cos 9tt (the cosine of the angle between 
the thrust axis of the candidate B and the thrust axis of the rest of the event), S (the sphericity), 
and cosOb (the cosine of the production angle of the candidate B, relative to the beam direction). 
In particular, T = R2 + 0.117| cos 6tt\ + 0.779(1 — S) + 0.104| cos with values ranging from 0.0 
to +2.0. The coefficients of all terms but R2 were determined by the standard Fisher discriminant 
procedure [Q . The relative weight given to R2 was determined visually, from a scatter plot of R2 vs. 
the Fisher discriminant from the other three variables. This Fisher discriminant is identical to the 
one we used in Ref. We apply loose cuts, 0.0 < T < 1.08. Our third and fourth variables used in 
the likelihood function are the signal-candidate B reconstruction variables conventionally used for 
decays from the T(45): beam-constrained mass Mcand = \/ -E'beam " ^cand and AE = ^cand-^beam- 
Our resolution in Mcand is 2.5 MeV, and in AE, 20 MeV. We apply loose cuts, 5.20 < Mcand < 5.30 
GeV and -0.25 < AE < +0.25 GeV. 

We thus have a likelihood function that depends on four variables: Mcand, AE, i^missj and 
We vary the branching fraction for the signal and the yields for the two backgrounds, to maximize 



^Throughout this article, the symbols tp and xp' mean J/tp{lS) and '0(2S), respectively. 
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the likelihood. Probability density functions (PDFs) are obtained from Monte Carlo samples of 
continuum events, BB events, and signal events. For signal events, lacking a compelling theoretical 
model, we use 3-body phase space, with final-state radiation as given by the CERNlib subroutine 
Photos [||]. 

Correlations among the four variables are weak, both for signal and backgrounds, and we 
ignore them. Distributions in the four variables, for signal and the two backgrounds, are shown for 
/x^ in Fig. ||. Distributions for B~^ h i~^£~^ are similar. 

For the decays whose quark-level process is 6 ^ se^/i^, we assume the branching fraction 
relations B{B- R-e^fi^) = B{B^ K^e^fi^) and B{B- K*-e^^i^) = B{B° K*^e^fi^), 
imposing the equalities as constraints in the maximum likelihood procedure. Thus our results 
here are for the average branching fraction B{B Ke^fi^) = 0.5{B{B^ — > K~e^fi^) + B{B^ — > 
K^e^^^)), and similarly with K* replacing K. For the decays whose quark-level process is 6 ^ 
de^jj^ , we assume B{B^ —>■ 7r°e^//^) = 0.5B{B~ —>■ 7r~e^^^), and similarly for the p'', p~ pair. 
Again, we impose those constraints in the maximum likelihood procedure, using information from 
both vr^ and vr*^ modes but quoting the "average" branching fraction B{B vre^/i^) = 0.5{B{B^ — > 
Tr~ e^fj.^) + 2B{B'^ — > vr'^e^/x^)), and similarly with the p*^, p~ pair. In all cases, by B{B — > he^jX^) 
we mean the sum B{B he'^ p") + B{B he^ p^). 

Our search is thus for four different lepton-flavor-violating final states: e^ji^, e^e^, e^/i^, 
and /i^/i'*'; with four different hadronic final states: K*, vr, p; a total of 16 decays. For each 
of the 16 decays, we maximize the likelihood C, as a function of signal branching fraction, by 
varying the yields of the two backgrounds. (In so doing, we constrain both backgrounds to be non- 
negative.) The central value obtained for signal is that giving the largest likelihood. The statistical 
significance of the signal is the square root of the difference in 21n£ between the maximum C 
and the C with signal branching fraction set to zero. If the largest likelihood corresponds to a 
negative signal, we assign a significance of zero. We find no compelling evidence for any of the 
decays. All but B K*e^p^ have a statistical significance of less than 1.2 standard deviations, 
while B K*e^p^ has a statistical significance of 2.0 standard deviations. In 16 searches, the 
probability that one of the 16 will fluctuate up by at least 2 standard deviations is ~l/3, so our 
result is consistent with all branching fractions being zero, and no claim for a signal is being made. 

We obtain 90% confidence level upper limits on the 16 branching fractions by integrating the 
likelihoods, as a function of the assumed branching fraction, from zero to that value which gives 
90% of the integral from zero to infinity. We increase the upper limit so found by 1.28 times the 
estimated systematic error, which includes contributions from uncertainty in efficiency for detecting 
the signal and uncertainty in the PDFs. The upper limits are increased by typically 12% from these 
systematic error considerations. Results are given in Table |. The limits on decays to vr, K range 
from 1.0 to 2.0 x 10~^, while those on decays to p, K* range from 2.6 to 8.3 x 10^®. 

As a check on the correctness of our continuum background PDFs, obtained from Monte Carlo, 
we have analyzed the off-resonance data, both alone and with 4 randomly chosen signal Monte 
Carlo events added. We found no evidence of 'signal' in the off-resonance data, and the correct 
amount of signal (average of 4.25, in 100 'toy experiments' for each of the 16 modes) when Monte 
Carlo signal events were added. 

We have performed two checks on the correctness of our BB background PDFs. In the first, 
we added 4 randomly chosen signal Monte Carlo events to the on-resonance data, and reanalyzed 
the data, performing 100 such 'toy experiments' on each of the 16 decay modes. We found an 
average of 4.0 signal events, in agreement with the number added. This check shows that whatever 
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bias is present in our analysis approximately cancels whatever real signal is present, an unlikely 
coincidence unless both are small. In the second check, we summed the on-resonance data sample 
for the 16 decay modes, and fitted it, with no signal allowed in the fit and with the continuum 
background constrained to the scaled off-resonance yield. In Fig. ^ we show the results of the fit 
for the distributions in Mcandj ^E, T, and -Emiss- Agreement is good. If instead we allowed signal 
in the fit and left the continuum background unconstrained (as in our actual analysis), we found 
4.01| q signal events for the sum over 16 modes. From these checks we conclude that any bias is 
small, ^ ^ event per mode, and is covered by our systematic error. 

In summary, we have searched for sixteen different lepton-flavor-violating decays of the form 
B hii. We find no evidence for any such decay, and place 90% confidence level upper limits on 
the branching fractions that range from 1.0 to 8.3 x 10~^. BaBar has limits on two of these decays 
1^], a factor of two more restrictive than ours. 

We gratefully acknowledge the effort of the CESR staff in providing us with excellent luminosity 
and running conditions. This work was supported by the National Science Foundation, the U.S. 
Department of Energy, the Research Corporation, and the Texas Advanced Research Program. 



Decay mode 


Significance 


Upper Limit 




of Signal 


(10-6) 


B Ke^pL^ 


O.Ocj 


1.6 




2.0(7 


6.2 




O.Ocj 


1.6 




0.6cr 


3.2 


B+ K'e+e+ 


O.Ocr 


1.0 


K*'e+e+ 


O.Ocj 


2.8 




O.Oa 


1.6 




l.lcr 


2.6 


B+ K-e+p+ 


O.Oa 


2.0 


K*-e+p+ 


O.Ocr 


4.4 




0.0(7 


1.3 


p~e~^p^ 


0.3a 


3.3 


B+ K-p+p+ 


O.Ocj 


1.8 


K*-p+p+ 


0.5cj 


8.3 




O.Ocj 


1.4 


P'P'^P'^ 


l.Ocj 


5.0 



TABLE I. For each of 16 decay modes, the statistical significance of the signal, and the 90% 
confidence level upper limit on the branching fraction, including systematic error. In the modes 
B — > he^p^, the limit quoted is on the sum B{B — > he^ p~) + B{B — > he" p^). 
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FIG. 1. Distributions in (a) Mcand, (b) AE, (c) and (d) E'miss for Monte Carlo samples of 
signal events (solid), BB background events (dotted), and continuum background events (dashed), 
for the search for B K^*^e^fi^ . The vertical scale is arbitrary. 
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FIG. 2. Results of the fit to the on-resonance data for the sum of the 16 modes with no signal 
allowed and the continuum background constrained to the scaled off-resonance yield. Distributions 
in (a) -Mcand) (b) A£', (c) T ^ and (d) i^miss- Points are on-resonance data; solid histogram is the 
fit. 



8 



REFERENCES 



[1] S. Glenn et al. (CLEO), Phys. Rev. Lett. 80, 2289 (1998), |hep-ex/9710003 . 

[2] T. Skwarnicki, Proceedings of the XXIX Int. Conf. on High Energy Physics, Vancouver, 

Canada, 1998, edited by A. Astbury, D. Axen, and J. Robinson (World Scientific, Singapore, 

1999), p. 1057. 

[3] S. Anderson et al (CLEO), Phys. Rev. Lett. 87, 181803 (2001), |liep-ex/0106060| . 
[4] T. AfFolder et al (CDF), Phys. Rev. Lett. 83, 3378 (1999), |hep-ex/990500^ . 
[5] B. Aubert et al (BaBar), [hep-ex/0201008| , submitted to Phys. Rev. Lett. 



[6] K. Abe et al (Belle), Phys. Rev. Lett. 88, 021801 (2002), |hep-ex/0109026 
[7] M. Sher and Y. Yuan, Phys. Rev. D 44, 1461 (1991). 
[8] S. Davidson, D. Bailey, and B. A. Campbell, Z. Phys. C 61, 613 (1994). 

[9] K. Zuber, Phys. Lett. B 479, 33 (2000), |hep-ph/0003T60| ; V. Gribanov, S. Kovalenko, and I. 
Schmidt, Nucl. Phys. B 607, 355 (2001), |hep-ph/0102T5E . 
[10] Y. Fukuda et al (Super-Kamiokande), Phys. Rev. Lett. 81, 1562 (1998), |hep-ex/ 9807003 . 



[11] H. V. Klapdor-Kleingrothaus, A. Dietz, H. L. Harney, and I. V. Krivosheina, Mod. Phys. Lett. 
A 16, 2409 (2001), |hep-ph/020l"23T . 



[12] Y. Kubota et al (CLEO), Nucl. Instrum. Methods Phys. Res., Sect. A 320, 66 (1992); T. Hill, 

Nucl. Instrum. Methods Phys. Res., Sect. A 418, 32 (1998). 
[13] G. Fox and S. Wolfram, Phys. Rev. Lett. 41, 1581 (1978). 

[14] R. A. Fisher, Ann. Eugen. 7, 179 (1936); M. C. Kendall and A. Stuart, The Advanced Theory 

of Statistics, Second Edition (Hafner, New York, 1968) Vol HI. 
[15] E. Barberio and Z. Was, Comput. Phys. Commun. 79, 291 (1994). 



9 



